Fix AT-SPI crash and screenshot hangs, add screen capture after sleep - #725
Open
mdshakib007 wants to merge 3 commits into
Open
Fix AT-SPI crash and screenshot hangs, add screen capture after sleep#725mdshakib007 wants to merge 3 commits into
mdshakib007 wants to merge 3 commits into
Conversation
…e after executing a sleep
mahbd
previously approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related reliability fixes to the Linux node and the screenshot path, all found while
investigating a node that was producing tracebacks instead of screenshots and periodically freezing
mid-run.
1. Node crash when AT-SPI is unavailable
Every screenshot on a Linux node without the AT-SPI stack produced this, and no screenshot:
Root cause.
pyatspiis a thin Python layer over the GObject-IntrospectionAtspitypelib, sothe pip side is only half of it — the system typelib (
at-spi2-core/gir1.2-atspi-2.0) and arunning accessibility bus are required too. Those are absent on a plain VM whether or not it has a
display. Three defects turned that expected absence into a hard failure:
sys.exit(1)at module import scope.sys.exitraisesSystemExit, which derives fromBaseException, notException— so theexcept Exception:guarding the import inCommonUtil.Thread_ScreenShotcould not catch it. It escaped to the bareexcept:inTakeScreenShot.capture_screenshotnever needed AT-SPI. It is built on xdotool / xwd / XComposite. A missingaccessibility stack was killing a screenshot path that does not use it.
each capture re-ran the import and re-launched the
uv adddependency-install subprocess.Changes
ATSPI_IMPORT_ERROR/XLIB_IMPORT_ERRORand exposed viais_atspi_available()/atspi_unavailable_message(), whichnames the required system packages.
pyatspifalls back to a stand-in whose attribute access raises a descriptiveRuntimeError, soAT-SPI actions fail as ordinary
zeuz_failedsteps instead of taking the node down.Action/EditableText/Textfall back toAny— theAccessibleprotocol evaluates them asannotations at class-creation time, so the names must exist.
except ImportErrortoexcept Exception: pyatspi raisesValueError: Namespace Atspi not availablewhen the typelib is missing, which the old code let crash.falls through to xwd, and
_get_frame_geometry_for_windowreturnsNonewithout AT-SPI so the fullwindow pixmap is kept rather than the capture being lost.
CommonUtil._get_linux_capture_screenshot()catchesBaseExceptionand caches the result, so theprobe runs once per process rather than per screenshot.
server/linux.py—if BuiltInFunctions is None:was unreachable dead code (the import aborted theprocess first). Now uses
is_atspi_available(), so/linux/inspectreturns the real reason.2. A screenshot could hang the node indefinitely
Runs would park on the
********** Capturing Screenshot ... **********line — sometimes for hours ordays — with no timeout able to recover them.
Root cause. Three facts combine into a permanent block:
TakeScreenShotis called outside_run_action_with_timeout, soaction_timeoutdoes notcover it and
_force_kill_hung_browser_sessionsnever fires.Thread_ScreenShotisasync def, butDriver.get_screenshot_as_file()is a synchronousSelenium HTTP call — it blocks the whole event loop.
RemoteConnection._timeoutdefaults tosocket._GLOBAL_DEFAULT_TIMEOUT,get_timeout()returnsNone,socket.getdefaulttimeout()isNone, and nothing in the codebase callsset_timeout. urllib3 waits forever.A wedged browser therefore blocked the node permanently. The log line is emitted before the capture,
which is why the run always stopped on exactly that line.
Change.
_capture_with_timeout()bounds every capture (SCREENSHOT_CAPTURE_TIMEOUT_SECONDS = 60):the same trade-off
_ActionTimeoutWorkeralready makes for a hung action.asyncio.wait_for.except WebDriverException/except Exceptionhandlers inThread_ScreenShotbehave exactly as before.The thread must be a daemon and must not use the default executor:
loop.shutdown_default_executor()joins default-executor threads, so an abandoned capture there would block node shutdown instead.
3. Screen capture after
sleepA
sleepis normally used to let a page settle, but the commonsleepaction declared"screenshot": "none"— so the moment you most wanted to see was the one moment never captured. Thepost-action capture at
sequential_actions.py:2789was already firing; it just had nothing to do.A common action cannot hardcode
"web", because it is shared by mobile, desktop and API runs — a fixed"web"would make every API-run sleep log a missing-driver warning.Change. New
"auto"screenshot type, declared forsleepand resolved at runtime inset_screenshot_vars()against the drivers the test actually has open:"web"when a Selenium orPlaywright driver is live,
"none"otherwise. Mobile is deliberately not resolved — an Appium captureon every sleep is expensive and wasn't needed here; it's a two-line branch in
_resolve_auto_screen_captureif wanted later.Behaviour changes
zeuz_failedsteps.Where AT-SPI is installed the code path is unchanged — every new line lives in an
exceptbranchthat previously ended in
sys.exit(1)./linux/inspectreturns a JSON error rather than killing the server process.sleep— slower runs and larger report ZIPs, by design.take_screenshot=falsestill disables it (sequential_actions.py:2734forces"none"first).thread stays parked on the dead connection until the browser is killed.
Testing
Verified on a real node run:
SystemExittraceback is gone; the node ran through the missing AT-SPI.screen_capture = "auto"→Capturing Screenshot for Action: Sleep Method: web.Automated:
tests/test_screenshot_capture_timeout.py(5 tests) — wedged capture times out; the event loop staysresponsive while a capture is stuck; success path; exceptions still propagate; Playwright awaitables
bounded too.
tests/test_auto_screen_capture.py(6 tests) — Selenium, Playwright, no-driver, torn-down browser(stale
Nonekeys), andnone/web/mobile/desktoppassing through untouched.test_nodejs_appium_installer).threadingis now actually used). No new mypy findings.Follow-ups (not in this PR)
Installer/setup_linux_inspector.shinstalls PyGObject's build deps, the X tools and the accessibilitysettings, but its apt branch never installs the AT-SPI runtime itself (
at-spi2-core,gir1.2-atspi-2.0); the dnf/Alma branch does installat-spi2-core-devel. That asymmetry is the likelyreason AT-SPI is unavailable on Ubuntu nodes.
install_missing_modulesrunsuv addat import time, which resolves the full dependency set and canattempt source builds during a test run. Worth removing from this path.
xdotoolfail every desktop capture; that is an environment fix (run the installerscript), not a code one.